{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XGBoost Parameter Tuning for Rent Listing Inqueries Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rental Listing Inquiries数据集是Kaggle平台上的一个分类竞赛任务，需要根据公寓的特征来预测其受欢迎程度（用户感兴趣程度分为高、中、低三类）。其中房屋的特征x共有14维，响应值y为用户对该公寓的感兴趣程度。评价标准为logloss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.参数调优（粗调）：max_depth & min_child_weight  \n",
    "(粗调，参数的步长为2；下一步是在粗调最佳参数周围，将步长降为1，进行精细调整)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用GridSearchCV进行两个参数的同时调优  \n",
    "注：GridSearchCV中最终判定的是scoring越大的模型越好，若用log_loss作为评价指标，在函数中要用neg_log_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "import xgboost as xgb\n",
    "\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>price</th>\n",
       "      <th>price_bathrooms</th>\n",
       "      <th>price_bedrooms</th>\n",
       "      <th>room_diff</th>\n",
       "      <th>room_num</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>...</th>\n",
       "      <th>walk</th>\n",
       "      <th>walls</th>\n",
       "      <th>war</th>\n",
       "      <th>washer</th>\n",
       "      <th>water</th>\n",
       "      <th>wheelchair</th>\n",
       "      <th>wifi</th>\n",
       "      <th>windows</th>\n",
       "      <th>work</th>\n",
       "      <th>interest_level</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.5</td>\n",
       "      <td>3</td>\n",
       "      <td>3000</td>\n",
       "      <td>1200.0</td>\n",
       "      <td>750.000000</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>4.5</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>5465</td>\n",
       "      <td>2732.5</td>\n",
       "      <td>1821.666667</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2850</td>\n",
       "      <td>1425.0</td>\n",
       "      <td>1425.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3275</td>\n",
       "      <td>1637.5</td>\n",
       "      <td>1637.500000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>18</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>3350</td>\n",
       "      <td>1675.0</td>\n",
       "      <td>670.000000</td>\n",
       "      <td>-3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>28</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 228 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   bathrooms  bedrooms  price  price_bathrooms  price_bedrooms  room_diff  \\\n",
       "0        1.5         3   3000           1200.0      750.000000       -1.5   \n",
       "1        1.0         2   5465           2732.5     1821.666667       -1.0   \n",
       "2        1.0         1   2850           1425.0     1425.000000        0.0   \n",
       "3        1.0         1   3275           1637.5     1637.500000        0.0   \n",
       "4        1.0         4   3350           1675.0      670.000000       -3.0   \n",
       "\n",
       "   room_num  Year  Month  Day       ...        walk  walls  war  washer  \\\n",
       "0       4.5  2016      6   24       ...           0      0    0       0   \n",
       "1       3.0  2016      6   12       ...           0      0    0       0   \n",
       "2       2.0  2016      4   17       ...           0      0    0       0   \n",
       "3       2.0  2016      4   18       ...           0      0    0       0   \n",
       "4       5.0  2016      4   28       ...           0      0    1       0   \n",
       "\n",
       "   water  wheelchair  wifi  windows  work  interest_level  \n",
       "0      0           0     0        0     0               1  \n",
       "1      0           0     0        0     0               2  \n",
       "2      0           0     0        0     0               0  \n",
       "3      0           0     0        0     0               2  \n",
       "4      0           0     0        0     0               2  \n",
       "\n",
       "[5 rows x 228 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# path to where the data lies\n",
    "dpath = './data/'\n",
    "train = pd.read_csv(dpath +\"RentListingInquries_FE_train.csv\")\n",
    "test = pd.read_csv(dpath +\"RentListingInquries_FE_test.csv\")\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>price</th>\n",
       "      <th>price_bathrooms</th>\n",
       "      <th>price_bedrooms</th>\n",
       "      <th>room_diff</th>\n",
       "      <th>room_num</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>...</th>\n",
       "      <th>virtual</th>\n",
       "      <th>walk</th>\n",
       "      <th>walls</th>\n",
       "      <th>war</th>\n",
       "      <th>washer</th>\n",
       "      <th>water</th>\n",
       "      <th>wheelchair</th>\n",
       "      <th>wifi</th>\n",
       "      <th>windows</th>\n",
       "      <th>work</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2950</td>\n",
       "      <td>1475.000000</td>\n",
       "      <td>1475.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2850</td>\n",
       "      <td>1425.000000</td>\n",
       "      <td>950.000000</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3758</td>\n",
       "      <td>1879.000000</td>\n",
       "      <td>1879.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>3300</td>\n",
       "      <td>1650.000000</td>\n",
       "      <td>1100.000000</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "      <td>4900</td>\n",
       "      <td>1633.333333</td>\n",
       "      <td>1633.333333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 227 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   bathrooms  bedrooms  price  price_bathrooms  price_bedrooms  room_diff  \\\n",
       "0        1.0         1   2950      1475.000000     1475.000000        0.0   \n",
       "1        1.0         2   2850      1425.000000      950.000000       -1.0   \n",
       "2        1.0         1   3758      1879.000000     1879.000000        0.0   \n",
       "3        1.0         2   3300      1650.000000     1100.000000       -1.0   \n",
       "4        2.0         2   4900      1633.333333     1633.333333        0.0   \n",
       "\n",
       "   room_num  Year  Month  Day  ...   virtual  walk  walls  war  washer  water  \\\n",
       "0       2.0  2016      6   11  ...         0     0      0    0       0      0   \n",
       "1       3.0  2016      6   24  ...         0     0      0    1       0      0   \n",
       "2       2.0  2016      6    3  ...         0     0      0    0       0      0   \n",
       "3       3.0  2016      6   11  ...         0     0      0    0       0      0   \n",
       "4       4.0  2016      4   12  ...         0     0      0    1       0      0   \n",
       "\n",
       "   wheelchair  wifi  windows  work  \n",
       "0           0     0        0     0  \n",
       "1           0     0        0     0  \n",
       "2           0     0        0     0  \n",
       "3           1     0        0     0  \n",
       "4           0     0        0     0  \n",
       "\n",
       "[5 rows x 227 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "发现测试集中没有y，这次作业无法得到在测试集上的表现情况"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*********** train **********\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 49352 entries, 0 to 49351\n",
      "Columns: 228 entries, bathrooms to interest_level\n",
      "dtypes: float64(9), int64(219)\n",
      "memory usage: 85.8 MB\n",
      "************ test ***********\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 74659 entries, 0 to 74658\n",
      "Columns: 227 entries, bathrooms to work\n",
      "dtypes: float64(9), int64(218)\n",
      "memory usage: 129.3 MB\n"
     ]
    }
   ],
   "source": [
    "print(\"*********** train **********\")\n",
    "train.info()\n",
    "print(\"************ test ***********\")\n",
    "test.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "测试集中少一维，即y标签"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAELCAYAAAARNxsIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGx5JREFUeJzt3X2wHXWd5/H3h/AgPibIhcokYYKaGUUdo14hypSj6EDAHRMpUChHIkNtxAKF0rUEd5aMPMyM4ygrM8pupogkrkOIiBI1GDMMyII8JGB4CJHlioxcyZJgeNQVivDZP/p35eTm5N7OpU9OTu7nVdV1ur/96z7fwy340v379a9lm4iIiCbs0e0EIiJi95GiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIas2e3E9jZ9t9/f0+fPr3baURE9JTbbrvtEdt9o7Ubd0Vl+vTprFmzpttpRET0FEn/Uaddbn9FRERjOlZUJL1I0q2S7pC0TtLnS/xSSb+QtLYsM0tcki6SNCDpTklvaTnXPEn3lWVeS/ytku4qx1wkSZ36PRERMbpO3v56GjjC9lOS9gJukHR12fcZ21cMa380MKMshwEXA4dJ2g9YAPQDBm6TtNz2o6XNfOBmYAUwG7iaiIjoio5dqbjyVNncqywjzbM/B1hSjrsZmChpMnAUsMr25lJIVgGzy76X277J1fz9S4C5nfo9ERExuo72qUiaIGktsJGqMNxSdl1QbnFdKGmfEpsCPNhy+GCJjRQfbBNvl8d8SWskrdm0adML/l0REdFeR4uK7S22ZwJTgUMlvQE4G3gt8DZgP+CzpXm7/hCPId4uj4W2+2339/WNOiIuIiLGaKeM/rL9GHAdMNv2hnKL62ng68ChpdkgMK3lsKnAQ6PEp7aJR0REl3Ry9FefpIllfV/gvcDPSl8IZaTWXODucshy4KQyCmwW8LjtDcBK4EhJkyRNAo4EVpZ9T0qaVc51EnBVp35PRESMrpOjvyYDiyVNoCpey2x/X9K/S+qjun21Fji1tF8BHAMMAL8FTgawvVnSecDq0u5c25vL+seBS4F9qUZ9ZeRXREQXqRo4NX709/c7T9RH7NoO/6fDu53Cbu/GT9y4Q+0l3Wa7f7R2eaI+IiIak6ISERGNSVGJiIjGpKhERERjUlQiIqIxKSoREdGYFJWIiGhMikpERDQmRSUiIhqTohIREY1JUYmIiMakqERERGNSVCIiojEpKhER0ZgUlYiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMR0rKpJeJOlWSXdIWifp8yV+sKRbJN0n6XJJe5f4PmV7oOyf3nKus0v8XklHtcRnl9iApLM69VsiIqKeTl6pPA0cYftNwExgtqRZwBeAC23PAB4FTintTwEetf0a4MLSDkmHACcArwdmA1+TNEHSBOCrwNHAIcCJpW1ERHRJx4qKK0+Vzb3KYuAI4IoSXwzMLetzyjZl/3skqcSX2n7a9i+AAeDQsgzYvt/2M8DS0jYiIrqko30q5YpiLbARWAX8HHjM9rOlySAwpaxPAR4EKPsfB17ZGh92zPbiERHRJR0tKra32J4JTKW6snhdu2blU9vZt6PxbUiaL2mNpDWbNm0aPfGIiBiTnTL6y/ZjwHXALGCipD3LrqnAQ2V9EJgGUPa/AtjcGh92zPbi7b5/oe1+2/19fX1N/KSIiGijk6O/+iRNLOv7Au8F1gPXAseVZvOAq8r68rJN2f/vtl3iJ5TRYQcDM4BbgdXAjDKabG+qzvzlnfo9ERExuj1HbzJmk4HFZZTWHsAy29+XdA+wVNL5wE+BS0r7S4BvSBqgukI5AcD2OknLgHuAZ4HTbG8BkHQ6sBKYACyyva6DvyciIkbRsaJi+07gzW3i91P1rwyP/w44fjvnugC4oE18BbDiBScbERGNyBP1ERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGpKhERERjUlQiIqIxKSoREdGYFJWIiGjMqEVF0ksk7VHW/0jS+yXt1fnUIiKi19S5UrkeeJGkKcA1wMnApZ1MKiIielOdoiLbvwWOBf7J9geo3gkfERGxlVpFRdLbgQ8DPyixTk6ZHxERPapOUTkTOBv4Tnm3yauoXrQVERGxlVGvOGz/GPixpJeU7fuBT3Y6sYiI6D11Rn+9vbytcX3ZfpOkr3U8s4iI6Dl1bn/9d+Ao4NcAtu8A3tnJpCIiojfVevjR9oPDQls6kEtERPS4OqO4HpT0DsCS9qbqT1nf2bQiIqIX1blSORU4DZgCDAIzy3ZERMRWRi0qth+x/WHbB9o+wPZf2v71aMdJmibpWknrJa2TdEaJ/42kX0laW5ZjWo45W9KApHslHdUSn11iA5LOaokfLOkWSfdJurxcSUVERJfUGf21WNLElu1JkhbVOPezwKdtvw6YBZwmaehJ/AttzyzLinLeQ4ATgNcDs4GvSZogaQLwVeBoqif5T2w5zxfKuWYAjwKn1MgrIiI6pM7trz+x/djQhu1HgTePdpDtDbZvL+tPUvXDTBnhkDnAUttP2/4FMAAcWpYB2/fbfgZYCsyRJOAI4Ipy/GJgbo3fExERHVKnqOwhadLQhqT92MFpWiRNpypEt5TQ6ZLulLSo5dxTgNZRZoMltr34K4HHbD87LB4REV1Sp6h8CfiJpPMknQf8BPiHul8g6aXAt4EzbT8BXAy8mqrDf0M5P4DaHO4xxNvlMF/SGklrNm3aVDf1iIjYQXU66pcAxwEPAxuBY21/o87Jy3tXvg180/aV5XwP295i+zngX6hub0F1pTGt5fCpwEMjxB8BJkrac1i83W9YaLvfdn9fX1+d1CMiYgzqvvnxZ8CVwFXAU5IOGu2A0udxCbDe9pdb4pNbmn0AuLusLwdOkLSPpIOBGcCtwGpgRhnptTdVZ/5y26aa2PK4cvy8kl9ERHTJqH0jkj4BLKC6UtlCddvJwJ+McujhwEeAuyStLbHPUY3emlnO8QDwMYAyA/Iy4B6qkWOn2d5ScjgdWAlMABbZXlfO91lgqaTzgZ9SFbGIiOiSOh3uZwB/XOfZlFa2b6B9v8eKEY65ALigTXxFu+PKjMmHDo9HRER31Ln99SDweKcTiYiI3lfnSuV+4DpJPwCeHgq29pNERERAvaLyy7LsXZaIiIi26rz58fMAkl5i+zedTykiInpV3vwYERGNyZsfIyKiMXnzY0RENCZvfoyIiMbkzY8REdGYEa9UyguyPmL7wzspn4iI6GEjXqmUubfm7KRcIiKix9XpU7lR0j8DlwO/f05l6K2OERERQ+oUlXeUz3NbYqZ6lW9ERMTvjdansgdwse1lOymfiIjoYaP1qTwHnL6TcomIiB5XZ0jxKkn/RdI0SfsNLR3PLCIiek6dPpW/Kp+tz6YYeFXz6URERC+rM0vxwTsjkYiI6H113lF/Uru47SXNpxMREb2szu2vt7Wsvwh4D3A7kKISERFbqXP76xOt25JeAXyjYxlFRETPqjX1/TC/BWaM1qiMFrtW0npJ6ySdUeL7SVol6b7yOanEJekiSQOS7pT0lpZzzSvt75M0ryX+Vkl3lWMukqQx/J6IiGhInTc/fk/S8rJ8H7gXuKrGuZ8FPm37dcAs4DRJhwBnAdfYngFcU7YBjqYqVjOA+cDF5fv3AxYAhwGHAguGClFpM7/luNk18oqIiA6p06fyjy3rzwL/YXtwtINsbwA2lPUnJa2nmj5/DvCu0mwxcB3w2RJfYtvAzZImSppc2q6yvRlA0ipgtqTrgJfbvqnElwBzgatr/KaIiOiAOkXll8AG278DkLSvpOm2H6j7JZKmA28GbgEOLAUH2xskHVCaTQFa3zA5WGIjxQfbxCMiokvq9Kl8C3iuZXtLidUi6aXAt4EzbT8xUtM2MY8h3i6H+ZLWSFqzadOm0VKOiIgxqlNU9rT9zNBGWd+7zskl7UVVUL5p+8oSfrjc1qJ8bizxQWBay+FTgYdGiU9tE9+G7YW2+2339/X11Uk9IiLGoE5R2STp/UMbkuYAj4x2UBmJdQmw3vaXW3YtB4ZGcM3j+U7/5cBJZRTYLODxcptsJXCkpEmlg/5IYGXZ96SkWeW7TqLeAIKIiOiQOn0qpwLfLC/qguoKoe1T9sMcDnwEuEvS2hL7HPD3wDJJp1D11xxf9q0AjgEGqIYtnwxge7Ok84DVpd25Q532wMeBS4F9qTro00kfEdFFdR5+/Dkwq/SNyPaTdU5s+wba93tA9VT+8PZm60krW/ctAha1ia8B3lAnn4iI6Lw6z6n8raSJtp8qQ4MnSTp/ZyQXERG9pU6fytG2HxvasP0o1W2qiIiIrdQpKhMk7TO0IWlfYJ8R2kdExDhVp6P+fwHXSPo61XMgf0X1JHxERMRW6nTU/4OkO4H3ltB5tld2Nq2IiOhFda5UAH4K7EV1pfLTzqUTERG9rM7orw8CtwLHAR8EbpF0XKcTi4iI3lPnSuW/Am+zvRFAUh/wb8AVnUwsIiJ6T53RX3sMFZTi1zWPi4iIcabOlcoPJa0ELivbH6KaUiUiImIrdUZ/fUbSscCfUk27stD2dzqeWURE9Jxao7/KtPVXjtowIiLGtfSNREREY1JUIiKiMdstKpKuKZ9f2HnpRERELxupT2WypD8D3i9pKcPejWL79o5mFhERPWekonIOcBbVu9+/PGyfgSM6lVRERPSm7RYV21cAV0j6b7bP24k5RUREj6rznMp5kt4PvLOErrP9/c6mFRERvajOhJJ/B5wB3FOWM0osIiJiK3UefnwfMNP2cwCSFlNNf392JxOLiIjeU/c5lYkt66/oRCIREdH76hSVvwN+KunScpVyG/C3ox0kaZGkjZLubon9jaRfSVpblmNa9p0taUDSvZKOaonPLrEBSWe1xA+WdIuk+yRdLmnvuj86IiI6Y9SiYvsyYBbV3F9XAm+3vbTGuS8FZreJX2h7ZllWAEg6BDgBeH055muSJkiaAHwVOBo4BDixtAX4QjnXDOBR4JQaOUVERAfVuv1le4Pt5bavsv1/ax5zPbC5Zh5zgKW2n7b9C2AAOLQsA7bvt/0MsBSYI0lUz8kMvShsMTC35ndFRESHdGPur9Ml3Vluj00qsSnAgy1tBktse/FXAo/ZfnZYvC1J8yWtkbRm06ZNTf2OiIgYZmcXlYuBVwMzgQ3Al0pcbdp6DPG2bC+03W+7v6+vb8cyjoiI2kYsKpL2aO1of6FsP2x7Sxme/C9Ut7egutKY1tJ0KvDQCPFHgImS9hwWj4iILhqxqJT/+N8h6aAmvkzS5JbNDwBDBWs5cIKkfSQdDMwAbgVWAzPKSK+9qTrzl9s2cC1wXDl+HnBVEzlGRMTY1Xn4cTKwTtKtwG+GgrbfP9JBki4D3gXsL2kQWAC8S9JMqltVDwAfK+daJ2kZ1RP7zwKn2d5SznM6sBKYACyyva58xWeBpZLOp3oY85I6PzgiIjqnTlH5/FhObPvENuHt/off9gXABW3iK4AVbeL38/zts4iI2AXUmVDyx5L+EJhh+98kvZjqqiEiImIrdSaU/M9Uz4P8zxKaAny3k0lFRERvqjOk+DTgcOAJANv3AQd0MqmIiOhNdYrK0+VpdgDKMN7tPhMSERHjV52i8mNJnwP2lfTnwLeA73U2rYiI6EV1ispZwCbgLqohwCuAv+5kUhER0ZvqjP56rkx5fwvVba97y8OHERERWxm1qEh6H/A/gJ9Tzbl1sKSP2b6608lFRERvqfPw45eAd9seAJD0auAHQIpKRERspU6fysahglLcD2zsUD4REdHDtnulIunYsrpO0gpgGVWfyvFUEz1GRERsZaTbX3/Rsv4w8GdlfRMwadvmEREx3m23qNg+eWcmEhERva/O6K+DgU8A01vbjzb1fUREjD91Rn99l2rK+u8Bz3U2nYiI6GV1isrvbF/U8UwiIqLn1SkqX5G0APgR8PRQ0PbtHcsqIiJ6Up2i8kbgI8ARPH/7y2U7Ypf0y3Pf2O0UxoWDzrmr2ynELqZOUfkA8KrW6e8jIiLaqfNE/R3AxE4nEhERva/OlcqBwM8krWbrPpUMKY6IiK3UKSoLxnJiSYuA/0Q1d9gbSmw/4HKqZ14eAD5o+1FJAr4CHAP8Fvjo0EAASfN4/v0t59teXOJvBS4F9qV6x8sZmZI/IqK7Rr39ZfvH7ZYa574UmD0sdhZwje0ZwDVlG+BoYEZZ5gMXw++L0ALgMOBQYIGkoSliLi5th44b/l0REbGTjVpUJD0p6Ymy/E7SFklPjHac7euBzcPCc4DFZX0xMLclvsSVm4GJkiYDRwGrbG+2/SiwCphd9r3c9k3l6mRJy7kiIqJL6rz58WWt25LmUl01jMWBtjeU826QdECJTwEebGk3WGIjxQfbxNuSNJ/qqoaDDjpojKlHRMRo6oz+2ort79L8Mypq91VjiLdle6Htftv9fX19Y0wxIiJGU2dCyWNbNvcA+hnhP+CjeFjS5HKVMpnnX/Y1CExraTcVeKjE3zUsfl2JT23TPiIiuqjOlcpftCxHAU9S9YGMxXJgXlmfB1zVEj9JlVnA4+U22UrgSEmTSgf9kcDKsu9JSbPKyLGTWs4VERFdUqdPZUzvVZF0GdVVxv6SBqlGcf09sEzSKcAvqd4iCdWQ4GOAAaohxSeX794s6Tyef9PkubaHOv8/zvNDiq8uS0REdNFIrxM+Z4TjbPu8kU5s+8Tt7HpPu5MBp23nPIuARW3ia4A3jJRDRETsXCNdqfymTewlwCnAK4ERi0pERIw/I71O+EtD65JeBpxBdVtqKfCl7R0XERHj14h9KuWJ9k8BH6Z6WPEt5SHEiIiIbYzUp/JF4FhgIfBG20/ttKwiIqInjTSk+NPAH1BN5vhQy1QtT9aZpiUiIsafkfpUdvhp+4iIGN9SOCIiojEpKhER0ZgUlYiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGdKWoSHpA0l2S1kpaU2L7SVol6b7yOanEJekiSQOS7pT0lpbzzCvt75M0rxu/JSIintfNK5V3255pu79snwVcY3sGcE3ZBjgamFGW+cDFUBUhYAFwGHAosGCoEEVERHfsSre/5gCLy/piYG5LfIkrNwMTJU0GjgJW2d5s+1FgFTB7ZycdERHP61ZRMfAjSbdJml9iB9reAFA+DyjxKcCDLccOltj24hER0SXbfUd9hx1u+yFJBwCrJP1shLZqE/MI8W1PUBWu+QAHHXTQjuYaERE1deVKxfZD5XMj8B2qPpGHy20tyufG0nwQmNZy+FTgoRHi7b5voe1+2/19fX1N/pSIiGix04uKpJdIetnQOnAkcDewHBgawTUPuKqsLwdOKqPAZgGPl9tjK4EjJU0qHfRHllhERHRJN25/HQh8R9LQ9/+r7R9KWg0sk3QK8Evg+NJ+BXAMMAD8FjgZwPZmSecBq0u7c21v3nk/IyIihtvpRcX2/cCb2sR/DbynTdzAads51yJgUdM5RkTE2OxKQ4ojIqLHpahERERjujWkuCe89TNLup3Cbu+2L57U7RQiokG5UomIiMakqERERGNSVCIiojEpKhER0ZgUlYiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGpKhERERjer6oSJot6V5JA5LO6nY+ERHjWU8XFUkTgK8CRwOHACdKOqS7WUVEjF89XVSAQ4EB2/fbfgZYCszpck4REeNWrxeVKcCDLduDJRYREV2wZ7cTeIHUJuZtGknzgfll8ylJ93Y0q+7aH3ik20nUpX+c1+0UdiU99bcDYEG7fwXHrZ76++mTO/y3+8M6jXq9qAwC01q2pwIPDW9keyGwcGcl1U2S1tju73YesePyt+tt+ftVev3212pghqSDJe0NnAAs73JOERHjVk9fqdh+VtLpwEpgArDI9roupxURMW71dFEBsL0CWNHtPHYh4+I2324qf7velr8fIHubfu2IiIgx6fU+lYiI2IWkqOwmMl1N75K0SNJGSXd3O5fYMZKmSbpW0npJ6ySd0e2cui23v3YDZbqa/wP8OdUw69XAibbv6WpiUYukdwJPAUtsv6Hb+UR9kiYDk23fLullwG3A3PH8716uVHYPma6mh9m+Htjc7Txix9neYPv2sv4ksJ5xPqtHisruIdPVRHSZpOnAm4FbuptJd6Wo7B5qTVcTEZ0h6aXAt4EzbT/R7Xy6KUVl91BrupqIaJ6kvagKyjdtX9ntfLotRWX3kOlqIrpAkoBLgPW2v9ztfHYFKSq7AdvPAkPT1awHlmW6mt4h6TLgJuCPJQ1KOqXbOUVthwMfAY6QtLYsx3Q7qW7KkOKIiGhMrlQiIqIxKSoREdGYFJWIiGhMikpERDQmRSUiIhqTohIREY1JUYkAJP2kRpszJb24w3nMHO05B0kflfTPDX9v4+eM8SlFJQKw/Y4azc4EdqiolNcS7IiZwLh+eC56W4pKBCDpqfL5LknXSbpC0s8kfVOVTwJ/AFwr6drS9khJN0m6XdK3yqSCSHpA0jmSbgCOl/RqST+UdJuk/y3ptaXd8ZLulnSHpOvLFDvnAh8qT2Z/qEbefZK+LWl1WQ6XtEfJYWJLuwFJB7Zr3/g/zBjX9ux2AhG7oDcDr6ealPNG4HDbF0n6FPBu249I2h/4a+C9tn8j6bPAp6iKAsDvbP8pgKRrgFNt3yfpMOBrwBHAOcBRtn8laaLtZySdA/TbPr1mrl8BLrR9g6SDgJW2XyfpKuADwNfLdz5g+2FJ/zq8PfC6F/jPK+L3UlQitnWr7UEASWuB6cANw9rMAg4BbqzmFGRvqvm7hlxejn8p8A7gW6UdwD7l80bgUknLgLHObvte4JCWc7+8vIHwcqqi9XWqCUYvH6V9RCNSVCK29XTL+hba/3siYJXtE7dzjt+Uzz2Ax2zPHN7A9qnlKuJ9wFpJ27SpYQ/g7bb/31bJSTcBr5HUB8wFzh+l/Ri+OmJb6VOJqO9JYOj/6m8GDpf0GgBJL5b0R8MPKC9s+oWk40s7SXpTWX+17VtsnwM8QvVOnNbvqONHVDNUU845s3yvge8AX6aalv3XI7WPaEqKSkR9C4GrJV1rexPwUeAySXdSFZnXbue4DwOnSLoDWAfMKfEvSrpL0t3A9cAdwLVUt6dqddQDnwT6Jd0p6R7g1JZ9lwN/yfO3vkZrH/GCZer7iIhoTK5UIiKiMemoj9hFSToZOGNY+Ebbp3Ujn4g6cvsrIiIak9tfERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGN+f+UQGoL+CmGNwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(train.interest_level);\n",
    "pyplot.xlabel('interest_level');\n",
    "pyplot.ylabel('Number of occurrences');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "总共有三类样本，每类样本分布的并不均匀，一般类别之间的数量差别在十倍以上时才考虑使用class_weight，这里暂不考虑"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop ids and get labels\n",
    "y_train = train['interest_level']\n",
    "\n",
    "train = train.drop([\"interest_level\"], axis=1)\n",
    "X_train = np.array(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第一轮参数调整得到的n_estimators最优值（338），除了需要调优的两个参数，其余参数继续默认值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': range(2, 12, 2), 'min_child_weight': range(1, 8, 2)}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#max_depth 建议3-10， min_child_weight=1／sqrt(ratio_rare_event) =5.5\n",
    "max_depth = range(2,12,2)\n",
    "min_child_weight = range(1,8,2)\n",
    "param_test2_1 = dict(max_depth=max_depth, min_child_weight=min_child_weight)\n",
    "param_test2_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面打印的结果以range的形式给出，等同于下方的取值  \n",
    "{'max_depth': [2,4,6,8,10], 'min_child_weight': [1,3,5,7]}  \n",
    "利用GridSearchCV将尝试5 * 4 = 20种参数的组合  \n",
    "\n",
    "Ps.虽然电脑很渣，但是想着要扔在那里跑一晚上，还是把参数范围设置大一点点好了，万一明天起来发现最佳值在边界的地方就傻了——2018/04/19  \n",
    "\n",
    "事实证明我是蠢的，误拿这里的训练分数logloss=0.5840979816337735和第一步参数调优的cv里train的logloss（0.488）进行对比了，以为这里怎么反而性能更差了，害我这个文件的训练调优跑了两次（10个小时左右），后来发现其实第一步参数调优cv里test的logloss是0.586325，这里的性能是有所提升的，但是由于之前误以为最佳学习器数量是337（实际是338），所以用337训练了两次，如果在用338训练一次时间太长了，暂时就直接用337训练，反正深度和权重的参数调出来后，还是要返回去调弱学习器数量的——2018/04/20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\model_selection\\_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.60754, std: 0.00302, params: {'max_depth': 2, 'min_child_weight': 1},\n",
       "  mean: -0.60738, std: 0.00302, params: {'max_depth': 2, 'min_child_weight': 3},\n",
       "  mean: -0.60723, std: 0.00305, params: {'max_depth': 2, 'min_child_weight': 5},\n",
       "  mean: -0.60748, std: 0.00315, params: {'max_depth': 2, 'min_child_weight': 7},\n",
       "  mean: -0.58728, std: 0.00297, params: {'max_depth': 4, 'min_child_weight': 1},\n",
       "  mean: -0.58726, std: 0.00347, params: {'max_depth': 4, 'min_child_weight': 3},\n",
       "  mean: -0.58697, std: 0.00302, params: {'max_depth': 4, 'min_child_weight': 5},\n",
       "  mean: -0.58711, std: 0.00281, params: {'max_depth': 4, 'min_child_weight': 7},\n",
       "  mean: -0.58519, std: 0.00338, params: {'max_depth': 6, 'min_child_weight': 1},\n",
       "  mean: -0.58410, std: 0.00377, params: {'max_depth': 6, 'min_child_weight': 3},\n",
       "  mean: -0.58487, std: 0.00371, params: {'max_depth': 6, 'min_child_weight': 5},\n",
       "  mean: -0.58431, std: 0.00346, params: {'max_depth': 6, 'min_child_weight': 7},\n",
       "  mean: -0.60001, std: 0.00461, params: {'max_depth': 8, 'min_child_weight': 1},\n",
       "  mean: -0.59590, std: 0.00489, params: {'max_depth': 8, 'min_child_weight': 3},\n",
       "  mean: -0.59508, std: 0.00393, params: {'max_depth': 8, 'min_child_weight': 5},\n",
       "  mean: -0.59216, std: 0.00413, params: {'max_depth': 8, 'min_child_weight': 7},\n",
       "  mean: -0.63369, std: 0.00440, params: {'max_depth': 10, 'min_child_weight': 1},\n",
       "  mean: -0.62032, std: 0.00496, params: {'max_depth': 10, 'min_child_weight': 3},\n",
       "  mean: -0.61063, std: 0.00401, params: {'max_depth': 10, 'min_child_weight': 5},\n",
       "  mean: -0.60749, std: 0.00265, params: {'max_depth': 10, 'min_child_weight': 7}],\n",
       " {'max_depth': 6, 'min_child_weight': 3},\n",
       " -0.5840979816337735)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb2_1 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=337,  #第一轮参数调整得到的n_estimators最优值\n",
    "        max_depth=5,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.5,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'multi:softprob',\n",
    "        seed=3)\n",
    "\n",
    "\n",
    "gsearch2_1 = GridSearchCV(xgb2_1, param_grid = param_test2_1, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch2_1.fit(X_train , y_train)\n",
    "\n",
    "gsearch2_1.grid_scores_, gsearch2_1.best_params_,     gsearch2_1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([ 284.42931037,  277.95901022,  278.86982312,  275.49055462,\n",
       "         485.64744663,  478.15741882,  482.63002076,  482.31681089,\n",
       "         695.19813275,  695.34783282,  692.36843252,  689.52872653,\n",
       "         914.29582748,  909.8462399 ,  906.91167035,  899.61937428,\n",
       "        1141.3790844 , 1129.26544843, 1113.39793777, 1098.00516429]),\n",
       " 'mean_score_time': array([0.66504817, 0.68566136, 0.63702841, 0.69466734, 1.04830418,\n",
       "        1.0517066 , 0.97825737, 0.99086552, 2.14564075, 2.09440713,\n",
       "        2.07499399, 2.08820205, 4.82203717, 4.91229863, 4.50762682,\n",
       "        4.50382481, 8.08242717, 7.39396467, 7.52024918, 6.44352665]),\n",
       " 'mean_test_score': array([-0.6075406 , -0.60738151, -0.60722976, -0.60747886, -0.58727609,\n",
       "        -0.58725857, -0.58696781, -0.58711102, -0.58518659, -0.58409798,\n",
       "        -0.58487448, -0.58430677, -0.60001452, -0.59589801, -0.5950841 ,\n",
       "        -0.59216257, -0.63369042, -0.62031593, -0.61063497, -0.60748667]),\n",
       " 'mean_train_score': array([-0.59392441, -0.59410647, -0.59431365, -0.59458186, -0.52347351,\n",
       "        -0.52645534, -0.52793768, -0.52968412, -0.40809059, -0.42294951,\n",
       "        -0.43290661, -0.43888948, -0.252374  , -0.29015773, -0.3143717 ,\n",
       "        -0.33234259, -0.12264495, -0.17316914, -0.20798708, -0.23348142]),\n",
       " 'param_max_depth': masked_array(data=[2, 2, 2, 2, 4, 4, 4, 4, 6, 6, 6, 6, 8, 8, 8, 8, 10, 10,\n",
       "                    10, 10],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'param_min_child_weight': masked_array(data=[1, 3, 5, 7, 1, 3, 5, 7, 1, 3, 5, 7, 1, 3, 5, 7, 1, 3,\n",
       "                    5, 7],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'max_depth': 2, 'min_child_weight': 1},\n",
       "  {'max_depth': 2, 'min_child_weight': 3},\n",
       "  {'max_depth': 2, 'min_child_weight': 5},\n",
       "  {'max_depth': 2, 'min_child_weight': 7},\n",
       "  {'max_depth': 4, 'min_child_weight': 1},\n",
       "  {'max_depth': 4, 'min_child_weight': 3},\n",
       "  {'max_depth': 4, 'min_child_weight': 5},\n",
       "  {'max_depth': 4, 'min_child_weight': 7},\n",
       "  {'max_depth': 6, 'min_child_weight': 1},\n",
       "  {'max_depth': 6, 'min_child_weight': 3},\n",
       "  {'max_depth': 6, 'min_child_weight': 5},\n",
       "  {'max_depth': 6, 'min_child_weight': 7},\n",
       "  {'max_depth': 8, 'min_child_weight': 1},\n",
       "  {'max_depth': 8, 'min_child_weight': 3},\n",
       "  {'max_depth': 8, 'min_child_weight': 5},\n",
       "  {'max_depth': 8, 'min_child_weight': 7},\n",
       "  {'max_depth': 10, 'min_child_weight': 1},\n",
       "  {'max_depth': 10, 'min_child_weight': 3},\n",
       "  {'max_depth': 10, 'min_child_weight': 5},\n",
       "  {'max_depth': 10, 'min_child_weight': 7}],\n",
       " 'rank_test_score': array([17, 14, 13, 15,  8,  7,  5,  6,  4,  1,  3,  2, 12, 11, 10,  9, 20,\n",
       "        19, 18, 16]),\n",
       " 'split0_test_score': array([-0.60269667, -0.60292467, -0.60248581, -0.60252954, -0.58235986,\n",
       "        -0.58220756, -0.58229182, -0.58296475, -0.57894279, -0.57821863,\n",
       "        -0.57895541, -0.57842067, -0.59344874, -0.58946899, -0.58822085,\n",
       "        -0.58550732, -0.62508035, -0.61075745, -0.60455748, -0.60240451]),\n",
       " 'split0_train_score': array([-0.5950179 , -0.59521566, -0.59520835, -0.59559081, -0.52537246,\n",
       "        -0.52910877, -0.53018159, -0.5318658 , -0.40915517, -0.42405239,\n",
       "        -0.43547111, -0.4394752 , -0.25216515, -0.29036272, -0.31492658,\n",
       "        -0.33530338, -0.12396735, -0.1768291 , -0.20792095, -0.2339889 ]),\n",
       " 'split1_test_score': array([-0.60718141, -0.60707631, -0.60655495, -0.60665405, -0.58647803,\n",
       "        -0.58698076, -0.58585459, -0.58663439, -0.58479314, -0.58312449,\n",
       "        -0.58342722, -0.58350016, -0.59763349, -0.59519404, -0.59515501,\n",
       "        -0.59109168, -0.63474344, -0.62030997, -0.60779067, -0.60834755]),\n",
       " 'split1_train_score': array([-0.59401357, -0.59457148, -0.59449518, -0.59466973, -0.52410634,\n",
       "        -0.52728144, -0.52808261, -0.53083478, -0.41092582, -0.42526734,\n",
       "        -0.43420992, -0.44143708, -0.2523961 , -0.28920162, -0.31362327,\n",
       "        -0.33120033, -0.12133316, -0.17300816, -0.20570423, -0.23213628]),\n",
       " 'split2_test_score': array([-0.60707589, -0.60612449, -0.60656638, -0.60679136, -0.58725071,\n",
       "        -0.58549142, -0.5871575 , -0.58612431, -0.58597871, -0.58277766,\n",
       "        -0.58481125, -0.58460304, -0.59866831, -0.59330953, -0.59483518,\n",
       "        -0.59125013, -0.63704624, -0.62328843, -0.61118265, -0.60883431]),\n",
       " 'split2_train_score': array([-0.59409727, -0.59397285, -0.59450131, -0.59467885, -0.52344643,\n",
       "        -0.52639621, -0.52831873, -0.52963901, -0.40863441, -0.4239282 ,\n",
       "        -0.43374994, -0.43891653, -0.25340807, -0.29260912, -0.31543594,\n",
       "        -0.33381239, -0.12476435, -0.17459541, -0.209692  , -0.23519658]),\n",
       " 'split3_test_score': array([-0.61208516, -0.6121077 , -0.61179071, -0.61192556, -0.58897531,\n",
       "        -0.58905178, -0.58794704, -0.58829315, -0.58775762, -0.58777424,\n",
       "        -0.58717465, -0.58607042, -0.6039785 , -0.59731479, -0.59697568,\n",
       "        -0.5955481 , -0.63493226, -0.62292921, -0.61452037, -0.60776538]),\n",
       " 'split3_train_score': array([-0.59312093, -0.59327172, -0.59348221, -0.59365427, -0.52213639,\n",
       "        -0.52422042, -0.52668606, -0.5280963 , -0.40680314, -0.4213768 ,\n",
       "        -0.43068517, -0.43675451, -0.25429337, -0.2910906 , -0.31548897,\n",
       "        -0.33210674, -0.12212117, -0.17103547, -0.20837867, -0.23270168]),\n",
       " 'split4_test_score': array([-0.60866419, -0.60867478, -0.60875142, -0.60949443, -0.59131775,\n",
       "        -0.59256292, -0.59158951, -0.59153982, -0.58846166, -0.58859626,\n",
       "        -0.59000543, -0.58894096, -0.60634547, -0.60420521, -0.60023533,\n",
       "        -0.59741722, -0.63665073, -0.62429579, -0.61512507, -0.6100824 ]),\n",
       " 'split4_train_score': array([-0.59337238, -0.59350062, -0.59388118, -0.59431564, -0.52230594,\n",
       "        -0.52526988, -0.52641942, -0.5279847 , -0.40493441, -0.4201228 ,\n",
       "        -0.43041688, -0.43786408, -0.24960731, -0.28752458, -0.31238376,\n",
       "        -0.32929011, -0.1210387 , -0.17037753, -0.20823957, -0.23338364]),\n",
       " 'std_fit_time': array([ 7.1780401 ,  4.98579446,  3.87033922,  4.52004833,  5.92074617,\n",
       "         2.65002476,  3.11674783,  4.36174645,  3.15464252,  3.06714025,\n",
       "         4.24089862,  4.39270023,  3.02386053,  3.5851294 ,  6.34476186,\n",
       "         4.15412655,  5.19845554,  3.39143936,  7.66281753, 10.23932671]),\n",
       " 'std_score_time': array([0.0291994 , 0.04019371, 0.00206027, 0.0699263 , 0.09105291,\n",
       "        0.1139148 , 0.00869124, 0.03237816, 0.03162106, 0.12774889,\n",
       "        0.16542991, 0.1661408 , 0.34379863, 0.54461179, 0.57105108,\n",
       "        0.32400284, 0.25288044, 0.25745829, 0.4434016 , 1.6986636 ]),\n",
       " 'std_test_score': array([0.00302412, 0.00301822, 0.00305026, 0.00314599, 0.00296739,\n",
       "        0.00346582, 0.00301514, 0.00280715, 0.00337946, 0.00376843,\n",
       "        0.00370804, 0.00346317, 0.00461258, 0.0048894 , 0.00393144,\n",
       "        0.00413139, 0.00440028, 0.00495725, 0.00401452, 0.00265335]),\n",
       " 'std_train_score': array([0.00066089, 0.00071109, 0.0005911 , 0.00062712, 0.00119652,\n",
       "        0.00168088, 0.00134707, 0.00151625, 0.00205361, 0.00189791,\n",
       "        0.00200599, 0.00157757, 0.0015781 , 0.00171868, 0.00120007,\n",
       "        0.00207868, 0.00147077, 0.00235801, 0.00129148, 0.00106078])}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch2_1.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.584098 using {'max_depth': 6, 'min_child_weight': 3}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAELCAYAAAAoUKpTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VPW9//HXZ5ZMdpaQkAkBsrBF2XewWBBRrBRr3cDlqtdqH/X6s7bVq15trb3WanvrUi+17dVar9a6XhWVquCGssgmm+wJCIFA2EMgmSQz398fc5LMJJNMtslk+Twfjzwyc873nPOdLPOe8/2e8/2KMQallFKqMbZoV0AppVTHp2GhlFIqLA0LpZRSYWlYKKWUCkvDQimlVFgaFkoppcLSsFBKKRWWhoVSSqmwNCyUUkqF5Yh2BdpKnz59TFZWVrSroZRSncratWuPGGNSw5XrMmGRlZXFmjVrol0NpZTqVETkm6aU02YopZRSYWlYKKWUCkvDQimlVFhdps9CKRVaZWUlhYWFlJeXR7sqKopiY2PJzMzE6XS2aHsNC6W6uMLCQpKSksjKykJEol0dFQXGGI4ePUphYSHZ2dkt2oc2QynVxZWXl5OSkqJB0Y2JCCkpKa06u9SwUKob0KBQrf0b6PZhYYzhq8V7KT2u7blKKdWQbh8WJ4vLWPlWPi/+fCVfvL6TstKKaFdJKaU6nG4fFj37xnPNg5MZPCGNjR/t44X7V7Dq3d1UlFVFu2pKdQknTpzgj3/8Y4u2feKJJzhz5kwb16htTJ8+vcWjRrz11lts2bKlWfvat28fM2bMIC8vj7PPPpsnn3yyRcduqW4fFgDJfeKYef1ZzPv5JAbk9Wb1u7t54f4VrF+yl6oKb7Srp1Sn1lXDojXqhkVTOBwOfv/737N161ZWrlzJggULmr2P1tBLZwP0zkhg9g9HUPxNCSvfLmDZ67tYv2QfEy7OYthUN3a7Zqvq3B5852u2HChp032elZHMA989u8H199xzD/n5+YwePZpZs2aRlpbGq6++isfj4dJLL+XBBx/k9OnTXHnllRQWFuL1evn5z3/OoUOHOHDgADNmzKBPnz588sknIfefmJjIv/3bv7FkyRJ69erFww8/zL//+7+zd+9ennjiCebOncuePXu47rrrOH36NAD//d//zdSpU3nzzTdZsGABixcv5uDBg3z7299m6dKlpKen1ztOWVkZN954I1u2bCEvL4+ysrKadR9++CEPPPAAHo+H3NxcnnvuORITE8nKyuKqq66qqftLL71EcXExCxcu5LPPPuOhhx7ijTfeAOC1117j1ltv5cSJEzz77LNMmzYt6Phutxu32w1AUlISeXl57N+/n7POOqsZv62W03e/ENIGJjP39tF87ydjSOrt4tO/b+cfv/ySHasPYnwm2tVTqlN55JFHyM3NZf369cyaNYudO3eyatUq1q9fz9q1a1m6dCnvv/8+GRkZbNiwgc2bNzN79mxuv/12MjIy+OSTTxoMCoDTp08zffp01q5dS1JSEvfffz+LFy/mzTff5Be/+AUAaWlpLF68mHXr1vHKK69w++23A3DppZeSnp7OggULuPnmm3nwwQdDBgXA008/TXx8PBs3buS+++5j7dq1ABw5coSHHnqIJUuWsG7dOsaPH89jjz1Ws11ycjKrVq3itttu44477mDq1KnMnTuX3/3ud6xfv57c3FwAqqqqWLVqFU888QQPPvggAAcOHOA73/lOvbrs2bOHr776ikmTJrXgN9IyembRiH5De/H9u8axZ9NRvnw7n8XPbmHdB3uZfEkOA4frdeuq82nsDKA9fPjhh3z44YeMGTMGgNLSUnbu3Mm0adO48847ufvuu5kzZ069T9WNiYmJYfbs2QCMGDECl8uF0+lkxIgR7NmzB/DfxX7bbbexfv167HY7O3bsqNn+qaeeYvjw4UyePJn58+c3eJylS5fWhMzIkSMZOXIkACtXrmTLli2cc845AFRUVDBlypSa7ar3OX/+fH7yk580uP/vf//7AIwbN66m3hkZGSxatCioXGlpKZdddhlPPPEEycnJYX8+bUXDIgwRIXtkH7KGp7BzzSG+fGc37y3YiDu3B5O/l0PG4F7RrqJSnYYxhnvvvZcf/vCH9datXbuWRYsWce+993LBBRfUnBWE43Q6az642Ww2XC5XzeOqKv+FKo8//jh9+/Zlw4YN+Hw+YmNja7bfv38/NpuNQ4cO4fP5sNkabnAJ9QHRGMOsWbP4xz/+EXabxj5gVtfbbrfX1LuuyspKLrvsMq655pqacGkv2gzVRGIThkxM5+pfTuLbVw+l5EgZb/7+K975w3oO7z0V7eop1WElJSVx6pT/f+TCCy/kr3/9K6WlpYD/jbq4uJgDBw4QHx/Ptddey5133sm6devqbdsaJ0+exO12Y7PZeOGFF/B6/ReuVFVVceONN/LSSy+Rl5cX1HxU17nnnsvf//53ADZv3szGjRsBmDx5MsuWLWPXrl0AnDlzJujM5ZVXXqn5Xn3G0ZLXZYzhpptuIi8vj5/+9KfN2rYtaFg0k91uY/i5/bj2P6cw9fuDOPRNCa8+vJr3/7KJ4wdPR7t6SnU4KSkpnHPOOQwfPpzFixdz9dVXM2XKFEaMGMHll1/OqVOn2LRpExMnTmT06NH8+te/5v777wfglltu4aKLLmLGjBmtqsOtt97K888/z+TJk9mxYwcJCQkAPPzww0ybNo1p06bx2GOP8cwzz7B169aQ+/jRj35EaWkpI0eO5Le//S0TJ04EIDU1lb/97W/Mnz+fkSNHMnnyZLZt21azncfjYdKkSTz55JM8/vjjAMybN4/f/e53jBkzhvz8/AbrHdhnsWzZMl544QU+/vhjRo8ezejRo+s1UUWSGNM1OmzHjx9vojFTnqesivVL9rJhyT6qKrwMneJmwsVZJKfEtXtdlApl69at5OXlRbsa3VL1DJ59+vSJdlWA0H8LIrLWGDM+3LbaZ9FKrjgHk76bw8jpmax9/xs2f7afHasOMnxaP8ZdlEV8cky0q6iUUq0W0bAQkdnAk4AdeMYY80iIMlcCvwQMsMEYc7WIDAT+z9rOCTxljPlTJOvaWnFJMXzrisGMmtmfNe/tZtNn+9myvIhR52UyZtYAXPEtG0NeKeU3adIkPB5P0LIXXniBESNGtOlxPvjgA+6+++6gZdnZ2bz55pvN3lf1VU1dQcSaoUTEDuwAZgGFwGpgvjFmS0CZwcCrwHnGmOMikmaMKRaRGKtuHhFJBDYDU40xBxo6XrSaoRpy4tAZvnyngF1rinHFOxh74UBGzMjEGWOPdtVUN6PNUKpaa5qhItnBPRHYZYwpMMZUAC8Dl9QpczOwwBhzHMAYU2x9rzDGVH+EcEW4nhHRs288F/5gOFf+xwTSc3qw4s18Xrx/BZs+LcRb5Yt29ZRSqlki+SbcD9gX8LzQWhZoCDBERJaJyEqr2QoAEekvIhutfTza2FlFR5Y6IIk5t43i0p+NpUdaHEtf3sFLv1zJ9pVF+PRucKVUJxHJsAh190ndd0cHMBiYDswHnhGRngDGmH3GmJHAIOB6Eelb7wAit4jIGhFZc/jw4TatfFvLGNyTS382ljm3jSImzsGSv23llYdWUbD+MF3lijSlVNcVybAoBPoHPM8E6p4dFAJvG2MqjTG7ge34w6OGdUbxNVDv/n9jzF+MMeONMeNTU1PbtPKRICIMHJ7ClfdO4MKbh+PzGv75p0288du1FG47Fu3qKaVUgyIZFquBwSKSbXVYzwMW1inzFjADQET64G+WKhCRTBGJs5b3As7BHyRdgtiEQePSmP+Licy4bhinT3h4+4n1vP3EVxza3bYjgioVbV11iPL2ns+imtfrZcyYMcyZM6dFx26piIWFMaYKuA34ANgKvGqM+VpEfiUic61iHwBHRWQL8AlwlzHmKJAHfCkiG4DPgP8yxmyKVF2jxWa3cdY5GVzzq8l864rBHCks5fVH17Do6Y0c3V8a7eop1Sa6ali0Rkvms6j25JNPRuXqtojeZ2GMWQQsqrPsFwGPDfBT6yuwzGJgZCTr1pE4nHZGzexP3jluNny0j/WL9/LyQ6sYOjGdCXOy6ZGqd4OrNvLPe+BgG3/uSh8BF9W7haqGzmfRNvNZABQWFvLee+9x3333NTqOVSR0uktSu7KYWAcTLs7muoemMub8AexaV8xLv1zJZ//YzumTnvA7UKoD0vks2m4+izvuuIPf/va3jY6MGyk63EcHFJvoZOplgxg1sz+rF+1hy+cH2La8iJHnZTLmgoHEJujd4KqFGjkDaA86n0XL57N49913SUtLY9y4cXz66adN/fG0GQ2LDiyhp4vpVw9lzKz+rHpnN+s+3MvmpQcYM2sAI8/LJCZWf32qc9H5LFo+n8WyZctYuHAhixYtory8nJKSEq699lpefPHFBvfZlrQZqhPokRrPrH89m3n3TyRjcE++XFjAiz9fwYaP9+Gt1LvBVcem81m0zXwWv/nNbygsLGTPnj28/PLLnHfeee0WFKBnFp1KSr9ELr51JAcLTrLyrXy+eHUn65fsZeKcbIZOSsdm1+xXHU/gfBYXXXRRzXwW4O+cfvHFF9m1axd33XUXNpsNp9PJ008/DdTOZ+F2uxvttwjn1ltv5bLLLuO1115jxowZIeezGD16NBMmTODiiy8OebXRj370I2688UZGjhzJ6NGjQ85nUT3Q4UMPPcSQIUOA2vksfD5fzdnHvHnzuPnmm/nDH/7A66+/3mC9Dxw4wA9+8IN2nbeiITqfRSdljKFw63FWvp1P8Ten6JUez8Tv5pA7NlXnBldBdCDB6NH5LFTUiQj9z+pNZl4vCtYf5su3C/jgfzaTOiCJyZfk0P+s3hoaSqk2o2HRyYkIuWPSyB6Vyo4vD7Lqnd2889QGMgb3ZPL3cnHn9oh2FZVqEzqfRXRpM1QX46308fUXB1jzzz2UlVSQNSKFSZfk0CczKdpVU1GizVCqmjZDqRp2p42RMzLJm+pm4yf7+OrDvbzy0GoGT+jLxDnZ9OwbH+0qKqU6IQ2LLsrpsjNudhZnT+vHV4v3svHjfexaW0zeOW4mfCeLxF6x4XeilFIWDYsuLjbByZTv5TJyRiZr//kNX3++n+0rDjJiej/Gzh5IXGJMtKuolOoE9ML8biKhh4tz5w3hmgcnM3h8Ghs+2scL969g1bu7qSivf7eoUm2lq446G40hyk+cOMHll1/OsGHDyMvLY8WKFS06fktoWHQzyX3imHnDWcz7+ST65/Vm9bu7eeH+FaxfspeqCm+0q6e6oK4aFq3R0iHKf/zjHzN79my2bdvGhg0b2vXCBQ2Lbqp3RgIX/XAEl98zntT+iSx7fRcv/mIlX3++H69XhxBRbSdwiPK77rqL3/3ud0yYMIGRI0fywAMPAP6RYy+++GJGjRrF8OHDeeWVV/jDH/5QM0T5jBkzGtx/YmIid999N+PGjeP8889n1apVTJ8+nZycHBYu9M+3tmfPHqZNm8bYsWMZO3Ysy5cvB+DNN9/k/PPPxxhDUVERQ4YM4eDBgyGPU1ZWxrx58xg5ciRXXXVVvSHKp0yZwtixY7niiitqhjPJysri7rvvZuLEiUycOJFdu3axfPlyFi5cyF133cXo0aPJz88H/EOUT5w4kSFDhvD555/XO35JSQlLly7lpptuAvwDKPbs2bO5v44W0z6Lbq5vVjJzfzyGwu3HWflWPp/+fTtfLd7LpO/mMGhcGmLTG/u6kkdXPcq2Y9vadJ/Deg/j7ol3N7j+kUceYfPmzaxfv54PP/yQ119/nVWrVmGMYe7cuSxdupTDhw+TkZHBe++9B/jHcurRowePPfYYn3zySaN3QFcPUf7oo49y6aWX1gxRvmXLFq6//nrmzp1bM0R5bGwsO3fuZP78+axZs4ZLL72UN954gwULFvD+++83eYjyjRs3MnbsWCB4iPKEhAQeffRRHnvssZqBEKuHKP/f//1f7rjjDt59913mzp3LnDlzuPzyy2v2Xz1E+aJFi3jwwQdZsmRJ0HAfBQUFpKamcuONN7JhwwbGjRvHk08+WTN0SaTpmYUCIHNoLy7793F850cjsDtsfPjs17zy8Gr2bDpCV7kXR0Vf4BDlY8eOZdu2bezcuZMRI0awZMkS7r77bj7//HN69Gj6zaR1hyj/9re/HXKI8ptvvpkRI0ZwxRVXBDUBPfXUU/zmN7/B5XKFHaL82muvBRoeonz06NE8//zzfPPNNzXbBQ5R3lgfQ7ghyquqqli3bh0/+tGP+Oqrr0hISOCRR9pvyHk9s1A1RITsUakMHNGHnasPseqdAt5bsBF3bg8mfy+HjMG9ol1F1UqNnQG0Bx2ivOVDlGdmZpKZmcmkSZMAuPzyy9s1LPTMQtVjswlDJ6Vz9YOT+fbVQzl5pIw3f/8V7zy1nsN7Wz9ctOpedIjythmiPD09nf79+7N9+3YAPvroI84666xm7aM19MxCNchutzH83H4Mm5zOpk/3s/aDPbz68Gpyx6YyaW4OvdLbp61UdW46RHnbDVH+1FNPcc0111BRUUFOTg7PPfdci38mzaVjQ6km85RVsX7xXtZ/tA9vhZdhU9xMmJNNUm+9G7wj07GhokeHKFfdkivOwaS5OYyYnsm6979h09JCtq86yPBz+zFudhbxyXo3uFJdlYaFarb45Bi+deVgRp3fn9Xv7WbTJ4VsWVbE6Jn9GT1rAK44/bNSbU+HKI8ubYZSrXb84GlWvbObXWuLccU7GHvhQEbMyMQZY4921RTaDKVqaTOUiqpe6QlcePNwxl54ipVvF7DizXw2fLyPCd/JIu+cDOwOvehOqc5Ow0K1mdQBSXz3/43iwM4TrHw7n8/+sYOvFu9l4ndzGDyhLza9G1ypTks/8qk2lzG4J5f+bCxzbhtFTJyDJc9t4ZWHVlGw/rDeDa5UJ6VnFioiRISBw1MYcFZvdq0r5suFBfzzT5vom53M5EtyyBzWO9pVVEo1g55ZqIgSmzB4fF+ufmASM64dxukTHt5+Yj1vP/EVh3aXRLt6qh101SHKozGfxeOPP87ZZ5/N8OHDmT9/PuXl5S06fktoWKh2YbPbOOtbGVzzq8l864rBHCks5fVH17Do6Y0cPVAa7eqpCOqqYdEaLZnPYv/+/fzhD39gzZo1bN68Ga/Xy8svvxyhGtanYaHalcNpZ9TM/lz30BQmfjeb/duP8/J/rmLJc1soOVIWfgeq09H5LNpmPgvwj2VVVlZGVVUVZ86cISMjo5m/jZaLaJ+FiMwGngTswDPGmHpDJIrIlcAvAQNsMMZcLSKjgaeBZMAL/NoY80ok66raV0ysgwkXZzPi25ms/eAbNn1ayM41hzj7WxmM+04WCT1c0a5il3Tw4YfxbG3b+SxcecNI/4//aHC9zmfRNvNZ9OvXjzvvvJMBAwYQFxfHBRdcwAUXXNDs31dLRSwsRMQOLABmAYXAahFZaIzZElBmMHAvcI4x5riIpFmrzgD/YozZKSIZwFoR+cAYcyJS9VXREZvo5JzLBjHqvP6sWbSbrz8/wNblRYw8rz9jLhhAbIIz2lVUbShwPguA0tJSdu7cybRp07jzzju5++67mTNnDtOmTWvyPuvOZ+FyuULOZ3Hbbbexfv167HZ70KiwTz31FMOHD2fy5Mlh57O4/fbbgYbnswCoqKioGSgRguez+MlPftLg/sPNZ3H8+HHefvttdu/eTc+ePbniiit48cUXa+bYiLRInllMBHYZYwoARORl4BIgsKHuZmCBMeY4gDGm2Ppe85s0xhwQkWIgFdCw6KISe7mYfs0wRs8awKp3drPuw2/Y+PE+csemMWyqm36De+qsfW2gsTOA9qDzWbR8PoslS5aQnZ1Namoq4A+X5cuXt1tYRLLPoh+wL+B5obUs0BBgiIgsE5GVVrNVEBGZCMQA+RGrqeoweqbFc8FNZzPv/okMnZzO7g2Hefvxr3jxFytY/d5uSo5qv0Zno/NZtM18FgMGDGDlypWcOXMGYwwfffRRuw7jEskzi1ARWveOLAcwGJgOZAKfi8jw6uYmEXEDLwDXG2N89Q4gcgtwC/h/kKrrSOmXyPRrhvGtKwZTsP4wW5cXseqd3ax6dzeZQ3uRN9VNzuhUHDr+VIen81m0zXwWkyZN4vLLL2fs2LE4HA7GjBnDLbfc0uKfSXNFbCBBEZkC/NIYc6H1/F4AY8xvAsr8CVhpjPmb9fwj4B5jzGoRSQY+BX5jjHkt3PF0IMGur+RoGdtXHmTr8iJOHS0nJs7B4Al9yZviJi0rqdFT/O5MBxKMHp3PomlWA4NFJBvYD8wDrq5T5i1gPvA3EemDv1mqQERigDeB/21KUKjuITkljgkXZzP+oiz27zzB1uUH2L6iiK+X7qeXO4G8qW6GTkrXeTWUioCIhYUxpkpEbgM+wH/p7F+NMV+LyK+ANcaYhda6C0RkC/5LZO8yxhwVkWuBc4EUEbnB2uUNxpj1kaqv6jzEJmQO7UXm0F545lWxa80hti4vYvkbu1j5Zj4DhqeQN9XNwBEp2O16K1FXofNZRJfOZ6G6jGNFp9m2vIjtXx7kTEkFcUlOhk5KZ9hUNykZidGuXtRoM5Sq1lGboZRqV73dCUy9bBCTv5fD3q+PsXV5ERs/LmT9kn2kDUwi75wMBo9PwxWv924o1VwaFqrLsdltZI3sQ9bIPpSdqmDHqkNsXX6Az17azhev7SRndCp5U91kDu2l924o1UQaFqpLi0uKYdTM/ow8L5PDe0+xdXkRO1cfYufqQyT2djFsspthU9z0SI2LdlWV6tA0LFS3ICKkDUwmbWAy51w+iN0bjrB1eRFr/rmHNYv20G9IT4ZNdZM7Jg2nS+/dUKouvVREdTsOp53B4/sy9/bR/MuvpzJpbg6njnv46G9bee7uL/jkha0cLDips/q1ka46RHk05rP413/9V9LS0hg+fHjQ8mPHjjFr1iwGDx7MrFmzOH78eIvq1RgNC9WtJfWOZfx3srj2V5O59GdjyB2dyo7Vh3jjt2t56Zdfsu6Dbzh90hN+R6pBXTUsWqMl81kA3HDDDbz//vv1lj/yyCPMnDmTnTt3MnPmTB55pN4A362mYaEU/maqjMG9mHnDWdz4228x47phxCU6WfFmPs/fu5x3F2wgf10x3qp6o86oMHQ+i7abz+Lcc8+ld+/6UxK//fbbXH/99QBcf/31vPXWW43+TlpC+yyUqiMm1sFZ52Rw1jkZnDh0hq0riti+ooj3Nx0lNtHJkIl9yZvqpk9mUrSr2myfv7qDI/vadmbCPv0TmXblkAbX63wWbTOfRWMOHTqE2+0GwO12U1xc3Gj5ltCwUKoRPfvGM+V7uUz6bjb7th5n6/IDbP5sPxs/LiR1QBLDprgZMrGvzrvRRDqfRcvns4g2DQulmsBmtzFweAoDh6dQXlrJjtX+AQ0/f2UHy97YSc6oVIZNddM/rze2DnzvRmNnAO1B57No+XwWjenbty9FRUW43W6KiopIS0sLv1EzaZ+FUs0Um+hk5Iz+XHXfRK68bwLDp/Vj37ZjvPvUBl64bzkr38rnRHHX65RtKZ3Pom3ms2jM3Llzef755wF4/vnnueSSS9ps39X0zEKpVkjtn0TqVUlM/f4gdm/037ux7oNvWPv+N7gH9SBvqpvcsWnExHbffzWdz6Jt5rMAf1PWp59+ypEjR8jMzOTBBx/kpptu4p577uHKK6/k2WefZcCAAbz2WtsP1q0DCSrVxkqPe9j+ZRFblxdxsrgMh8vOoHFp5E1x4x7Uo93n3dCBBKNH57NQSjUosZeLcbOzGHvhQA7mn2TriiJ2rSlm2/IieqTFMWyKm2GT00nsFRt+Z0p1EBoWSkWIiOAe1BP3oJ5Mu3II+euK2bq8iC/fLmDVwgL6n9WbYVPc5IxKxe7U7sNwdD6L6GpWM5SI9AL6G2M2Rq5KLaPNUKqzOHn4DNtWHGTbiiJKj3twxTsYMjHdf+9G/8Q2b6bSZihVLaLNUCLyKTDXKrseOCwinxljftqy6irVvfVIjWfS3BwmzMmmcNsxti0vYssXB9j0aSEp/RLJm+pmyKS+xCW23fSwxhido7yba23/dFOaoXoYY0pE5AfAc8aYB0Skw51ZKNXZ2GzCgLNSGHBWCuWnK9m5+hDbVhTxxWs7Wf5/u8ge2YdhU90MOKs3tlZMDxsbG8vRo0dJSUnRwOimjDEcPXo06P6S5mpKWDhExA1cCdzX4iMppRoUm+BkxPRMRkzP5Oj+UrYuL2LHqoPkf3WY+B4xDJuczrApbnqlJzR735mZmRQWFnL48OEI1Fx1FrGxsWRmZrZ4+7B9FiJyBfBz4AtjzK0ikgP8zhhzWYuPGgHaZ6G6Gm+Vj282HWXriiK+2XwU4zOk5/jv3Rg0Lo2YOL0+RbVeU/ss9D4LpTqB0yc9bP/yINuWF3H84BkcMTZyx/rv3cgY3FOnh1Ut1mZhISK/BR4CyoD3gVHAHcaYF9uiom1Fw0J1B8YYDu0u8d+7sfoQFeVekvvE+u/dmOImqbfeu6Gapy3DYr0xZrSIXAp8D/gJ8IkxZlTbVLVtaFio7qaywkvBV4fZuryI/duPg0D/Yb0YNtV/74YjRqeHVeG15R3c1WMvfwf4hzHmmF5RoVT0OWPsDJ2UztBJ6ZQcKWPbiiK2rTjI4me3EBPnYMiEvgyb6iZtYJJeBaVarSlh8Y6IbMPfDHWriKQC5ZGtllKqOZL7xDHxuzlMuDibwh3H2ba8iK0riti8dD+9MxL8925MTCc+ue3u3VDdS5M6uK07t0uMMV4RiQeSjTGh5x6MEm2GUiqY50wlO9cUs21FEYd2l2CzCQNHpJA31c2A4SnYW3Hvhuo62vIObidwHXCudSr7GfCnVtdQKRVRrngnw8/tx/Bz+3HswGn/9LBfHmT3hiPEJTkZOimdYVPdpGQkRruqqhNoSgf3M/j7LZ63Fl0HeI0xP4hw3ZpFzyyUCs/r9bF381G2Li/im01H8fkMaVnJ5E11M3h8Gq54nR62u2nLq6E21L3yKdSyaNOwUKp5zpRUsGOVf3rYYwdOY3fayBmdSt45bjKH9NJ7N7qJtrxneRgMAAAebUlEQVQayisiucaYfGvHOYC3tRVUSkVXfHIMo88fwKiZ/Sn+5hTblhexY/Uhdq4+RFLvWIZOSSdvipvkPnHRrqrqAJpyZjETeA4oAAQYCNxojGn5HIcRoGcWSrVeVYWXgg2H2ba8iH3bjoOBfkN7kjc1g5wxqTj13o0up02H+xARFzAUf1hsA0YbY75swnazgScBO/CMMeaREGWuBH4JGGCDMeZqa/n7wGT8Y1LNCXcsDQul2tapY+XWvRtFlBwpJybWTu64NNIGJJGUEkdS71iSUmJxujRAOrOIjg0lInuNMQPClLEDO4BZQCGwGphvjNkSUGYw8CpwnjHmuIikGWOKrXUzgXjghxoWSkWP8RkO7DrB1uVF5H91mCpPcCt0XJKzJjiqQyQ5pfp5LDGxOuBhRxbpObib0vM1EdhljCmwKvQycAmwJaDMzcACY8xxgOqgsB5/JCLTW1g/pVQbEZvQb0gv+g3pxcx/MZw+WcGpo2WcOlZOydFyTh0t59Sxco7uP82ejUfxVvmCtnclOEgOOBNJSon1B0off7i4dPTcTqGlv6WmnI70A/YFPC8EJtUpMwRARJbhb6r6pTHm/RbWSSkVYWITEnu5SOzlwh1ivfEZzpyq8AeIFSL+QCnj+MHT7P36KFWVwWESE+cgKcU6G7ECJTklriZYXPEOHa6kA2gwLETkHUKHggApTdh3qN9u3f05gMHAdCAT+FxEhhtjTjRh/4jILcAtAAMGNNoqppRqB2ITEnq4SOjhIj2nR731xhjKTlVy6pg/TEqOllF6tJySY+WcPFzGvm3H6zVzOWPtAU1btWcoyX384RKb6NQwaQeNnVn8VwvXVSsE+gc8zwQOhCiz0hhTCewWke34w2N1E/aPMeYvwF/A32fRlG2UUtEjIsQnxxCfHEPfrOR6640xeE5XWWckZTVnKNXNXQd2nqCiPDhMHDG2kH0l1WcocUkaJm2hwbAwxnzWyn2vBgaLSDawH5gHXF2nzFvAfOBvItIHf7NUQSuPq5TqpESE2EQnsYlOUgckhSzjOVNZ21cS1NxVxqHdJ/GcqQoqb3faGgySpN6xxCfH6A2ITRCxniVjTJWI3AZ8gL8/4q/GmK9F5FfAGmPMQmvdBSKyBf+NfncZY44CiMjnwDAgUUQKgZuMMR9Eqr5Kqc7BFe8kNd5Jav/QYVJRVhXc+V59hnKsnOK9pygvrQwqb3OIv2krsKmrJlBiie/hwqZhotOqKqW6l4ryKkqPeWqbuY4FNHUdK6espCKovM0mJPZ21YRIcsAVXUkpsST2dGHrxCP4RvrSWaWU6pRiYh30znDQOyMh5PrKCi+lx4L7SvyBUsber49y5mRwmIhNSOzpqgmSxJTgzvjE3q4uMRx8U4YoD3VV1ElgDfBnY4xOhKSU6jKcMXZ6pSfQKz10mFRVeik95gnqK6l+XLj9OKUnPEHvmCKQYIVJYF9JTf9Jr1jszo4fJk05sygAUoF/WM+vAg7h74z+H/xDliulVLfgcNrp2Teenn3jQ673VvkoPe7h1NGymqat6o74op0n2bnqEEGt/wIJyTH1+kpqAqV3bIeYT70pYTHGGHNuwPN3RGSpMeZcEfk6UhVrL1VeHzsOlWKzgSDYxH9FhgjYRBCs70LtsoDvgds0adugZdppplRXY3fY6JEaR4/U0KP1er0+Th/31OsrOWVdzZW/thifL7gxJy45JqivpO49J+0xPldTwiJVRAYYY/YCiMgAoI+1rqLhzTqHE2WVfOcPn0ft+EFBg9QPJKygsdUPGps0EHB1tw1YhrWPwHCkZl/WOn8K1u7fFrxt9b78F4hUH7/uttX1JcQy/z5Dbht0jNrXJXXKSp3gbaxs3XqH/vkRYp/SwPLAukuDr7H6Q0L9DxSNHAup88EldFkI9TdRu11Q3Ztcr5Z/eDHG4DPg9Rl8pvrL/9wYYy2nZp1/eejyPmPw+fxlvaZ6e2tbaz/e6m18dY5j7cfnq3OcEOVD15OA5f7nDdU/6DgmxOvxEVD/cNuDcRq8aQZvig1XJcRWGGIrDfGVhvjKCuKKKkjYd5KEKrDXuee5JNHGvf81vcW/v6ZoSlj8DPhCRPLxv59kA7eKSAK1s+d1WkmxDv507TjA/8szAb9Q8D/2L/M/xvpuoOYPDVP7h1C9PQHrg5ZZfySGENsSfIx621r1qq4P1P5Thdy23rLabU3ANiag7sbap6n5efgw3sBjWNuaBratuz7gGPW2ofZNJvC7CXoNtdsG7r/OBy/VRqSxDw1WoIR6E+4iF1XWEAG79TOo/rBkt4LWbqte7v+Z2K0PL3bruc1a31j56v06HDbsNmv76g9N1eVqlgtegTM2oVyEE4Cj0uAs9xHj8WEv89E7OSbiP5OwYWGMWWSNDjsMa4jygE7tJyJZufbgctiZPTw92tVQLdBYKNULlpDlgoPb1A21OmVDhmvA9vXK+uqGeIhj1YRz9fYBgR6mbEOvqfrY1R8YGv451T9W9YeXwNeECXzdYLfRRm+W9d8Ug7a3EbA84E27+jgiNc+DjiO1+21RPbWJOKSmXA3lBH4IVPdbfCoif7aG6FAqamqaWJo0CLJSqjWa0gz1NOAE/mg9v85a9oNIVUoppVTH0pSwmGCMGRXw/GMR2RCpCimllOp4mnIniFdEcqufiEgO/nGclFJKdRNNObO4C/hERArwd3APBG6MaK2UUkp1KE25Guoj62qooVhXQwGjI10xpZRSHUeTBhI0xniAjdXPReQ1QKemU0qpbqKlo1fptYpKKdWNtDQsutj9mkoppRrTYDNUA0OTg/+sIiViNVJKKdXhNNZn8V8tXKeUUqqLaTAsjDGftWdFlFJKdVwdf3ompZRSUadhoZRSKqxmhYWI6FjeSinVDTX3zGJRRGqhlFKqQ2tuWOjNeEop1Q01Nyz+JyK1UEop1aE1KyyMMX8MX0oppVRXo1dDKaWUCkvDQimlVFgaFkoppcLSsFBKKRWWhoVSSqmwIhoWIjJbRLaLyC4RuaeBMleKyBYR+VpEXgpYfr2I7LS+ro9kPZVSSjWuSdOqtoSI2IEFwCygEFgtIguNMVsCygwG7gXOMcYcF5E0a3lv4AFgPP45NdZa2x6PVH2VUko1LJJnFhOBXcaYAmNMBfAycEmdMjcDC6pDwBhTbC2/EFhsjDlmrVsMzI5EJY0xlG3YgLekJBK7V0qpLiFiZxZAP2BfwPNCYFKdMkMARGQZYAd+aYx5v4Ft+0Wikt5jx9hz1TwAHKmpxOTm4srNJSY3B1dOLq5BudhTUhDRkU6UUt1XJMMi1Ltr3WlaHcBgYDqQCXwuIsObuC0icgtwC8CAAQNaVElbfDyZT/+Rivx8PPkFePLzOfnWW/hOn64t06MHrtxcXLk5xFgB4srJweF2Iza9RkAp1fVFMiwKgf4BzzOBAyHKrDTGVAK7RWQ7/vAoxB8ggdt+WvcAxpi/AH8BGD9+fKj5wsOyxcWRNGMGzJgRuF+qiovx7NpFhRUgFfn5nProY7yvvV5TTuLjcWVn+89Ccgf5wyQ3l5j+/RFHJH+0SinVvsSYFr3Hht+xiAPYAcwE9gOrgauNMV8HlJkNzDfGXC8ifYCvgNFYndrAWKvoOmCcMeZYQ8cbP368WbNmTUReS6Cq48f9ZyG78vEU5NeESdXBgzVlxOkkJmsgMbmDcOXk4BqU6w+RrCxsLlfE66iUUk0lImuNMePDlYvYx19jTJWI3AZ8gL8/4q/GmK9F5FfAGmPMQmvdBSKyBfACdxljjlov4D/xBwzArxoLivbk6NULx/jxxI8P/tl6S0upKCjAk19ARYE/TMq3buHUhx+Cz+cvZLPh7J9Z0xcSk1PbtGVPTIjCq1FKqaaJ2JlFe2uvM4vm8nk8VOzZE3A2UkBF/i48e76Bysqaco709Hod6zG5uTh69Ypi7ZVSXV3UzyyUn83lInboUGKHDg1abqqqqNi7z38Wkl+AJ9/fP3LitdcxZWU15ey9e+PKyQm+SmvQIBxpaXqFllKq3WhYRIk4HLhysnHlZJN0fu1y4/NRVVSEp6AAz678miatkvffx3fyZE05W2IiMTk5tVdpWWHi7NcPsduj8IqUUl2ZNkN1EsYYvEeP1nas1zRp5VN1+HBNOXG5iMnO9p+NDMr1N2nl5hAzcCASExPFV6CU6oi0GaqLEREcffrg6NOHhMnB9zZ6S0pqLu/15BfgKcinbMMGShYtqi1ktxMzYEBwx3puLq7sbGzx8e38apRSnY2GRRdgT04mfswY4seMCVruO3MGz+7d1lVatWFy6uNPwOutKefs1y+4Y90KE3uPHu39UpRSHZSGRRdmi48n7uyziTv77KDlpqKCir17gzrWPfn5nPlyFcbjqSlnT+1jNWPlBt14aO/TRzvXlepmNCy6IYmJwTVoEK5Bg4ALapYbr5fKAwf8d65bHeyegnxOLlyIr7S0ppwtOblegMTk5OLM0OFPlOqqtINbhVU9/EmoO9e9x2rvlZS4OP/wJ1bHenWYxAzQ4U+U6qi0g1u1GRHB2bcvzr59SZg6NWhdzfAnAXeun1m9hpKF79QWcjpxZQ0M7ljPzSUmO1uHP1Gqk9CwUK3S8PAnp6nYHdyxXr5tK6cWLw4e/iQzs3b8rJoO9hzsiYlReDVKqYZoWKiIsCcmEDdiBHEjRgQt9w9/8o1/yJOAEX1PL1uGCRz+pG9fnBkZSKwLW4wLcbn8j10upPq5KwabKzbgsbUuqJy1PDYWiXFhc8VY5V1ITIx21CvVRBoWql35hz8ZQuzQIUHLTVUVFfv21XSsVxTkU3nwEKbcQ9XJEnwVHky5B+Px4KuowHg8mPLyVtenJjhcMcGhFBRILsQVGxxILhe22LrBVaecVdYWG/A4MKz0YgDViWhYqA5BHA5c2dm4srNJmjmzSdsYYzCVlf7gsL58ngpMhT9Iah57PPg8VthUWI89VuBUePBZIeRfZy33+Lf3nT5NlcfjDytPhX+/Ff7vNc1pLX3NTmf9sIqNDQ4ul6s2eBo8e7LWWWdPDYaVy4UtxgorveBANZP+xahOS0T8Q5jExEBSUrsf31RV1QZRU8LKU2GFUFPCyoO3pKR2v3XCKnDE4hZxOGqDI1STXsizLH/wOFLTcLjTcaa7cWa4sffurc153YCGhVItJA4H4nBgS2j/uUiM14uxmuN8VggFhZXHg6+83B8wdc6mfJ7y4LAKcdblO3MG34njAU1/1vKysnpBJTExteGRno4jw10TJM70dBxut16w0AVoWCjVCYndjsTFQVwc7TnGsDEG7/HjVBYVUXXwIJUHiqg8WERV0UEqi4o4vWoVVYcO1WuisyUlBQeJOx2n243DeuxIT8emA112aBoWSqkmExEcvXvj6N0b6gwjU81UVVF1+DCVRQepLDoQECoHqSoqonzjJrzHj9fbzt6nD0539dlIOk53RlCoOFL76EUBUaRhoZRqU+Jw+N/03W5gTMgyvvJyf4gUFQWHStFBPAUFnF62DN+ZM8EbORw4+/atbfJyu61QcdeEjK1HD+0/iRANC6VUu7PFxhKTlUVMVlbI9cYYfCUlVFqBUlUdKgeLqDpQRNn69ZR88EH9/pP4eJzp6XWavKqbuvzfbXFx7fAKux4NC6VUhyMi2Hv0wN6jR70piasZn4+qI0fq9J1Uh8pByj/7DO/hI/W2s/fsWRsk6ek4M6y+k+oO+bQ0vbQ4BP2JKKU6JbHZcKal4UxLI27kyJBlTEUFlcXFVB44UL9DvrCQM2vW4CspCd7IZsORlhbcdxIYKu70bnm5sIaFUqrLkpgYYjIzicnMbLCMt/Q0VQfr951UFhXh2bKV0o8+xlRUBO/X5cKR3rfBvhOHOwN7YvtfUh1JGhZKqW7NnpiAvWZ+l/qCLhcO7DuxHp9euZKq4uLQlwtXB0lg34kVKo6+fTvV5cIaFkop1YjmXS5cVCdU/GcrDV4unNqnft9JQId8R7pcWMNCKaVaKfhy4dB8ZWX+e00C+06sx56CAkqXLcPUvVzY6fT3y7jd/jOSmlCpbfayJSe3S/+JhoVSSrUDmzWTpCs7O+T6oMuFDwT3nVQVFVG2bh0lxcUhLxdOmDyZ/n9cENH6a1gopVQH0KzLhev0ndh79Yp4/TQslFKqkwi6XHjUqHY9dsfoOVFKKdWhaVgopZQKS8NCKaVUWBENCxGZLSLbRWSXiNwTYv0NInJYRNZbXz8IWPeoiGy2vq6KZD2VUko1LmId3CJiBxYAs4BCYLWILDTGbKlT9BVjzG11tr0YGAuMBlzAZyLyT2NMnUFclFJKtYdInllMBHYZYwqMMRXAy8AlTdz2LOAzY0yVMeY0sAGYHaF6KqWUCiOSYdEP2BfwvNBaVtdlIrJRRF4Xkf7Wsg3ARSISLyJ9gBlA/7obisgtIrJGRNYcPny4reuvlFLKEsmwCHX/uanz/B0gyxgzElgCPA9gjPkQWAQsB/4BrACq6u3MmL8YY8YbY8anpqa2Zd2VUkoFiGRYFBJ8NpAJHAgsYIw5aozxWE//BxgXsO7XxpjRxphZ+INnZwTrqpRSqhGRDIvVwGARyRaRGGAesDCwgIgEjro1F9hqLbeLSIr1eCQwEvgwgnVVSinViIhdDWWMqRKR24APADvwV2PM1yLyK2CNMWYhcLuIzMXfxHQMuMHa3Al8bo2kWAJca4yp1wyllFKqfYgxdbsROqfx48ebNWvWRLsaSinVqYjIWmPM+HDl9A5upZRSYWlYKKWUCkvDQimlVFgaFkoppcLSyY8qy2D9S6HXNTivbYjlzSnbYPnmlI3CvtukHg3tooE62+wg9oDvtoDntkbWBZSpfh74uO66oO+Rn89Yqc5Gw6LiNLz302jXQnUo0rIwCtqmKWWrl4Uq28g+QoZcA/uQUIEbJijD1cUYMD7AWI+N9dgXYp2vkXUNbdecsg1tR+3zxo5R85gwx6+7zjThNfrqlGvs+CFeb3N+jqlDYM7jEf2v0LCI6w0/2xFiRQOXFIe81Lg5ZRso35yyUdt3e9fDBz4vGK/1PeB50DpfQBlvw8trtqm73xD7r37eZvv3ga8Kqjx19t9A2VDHMybEMl/Tf1eqGcQ6wxQroK3vgc9rHlN/Xc3zutvRyLoQZZt0fOtxhGlY2GyQ1DfatVCqZao/ZYYMP9O0kGssjEKFX6g3q6A3wDBvss0pW28dYY7fnGM0sJ02Q4akYaFUZxbYbKRUBOnVUEoppcLSsFBKKRWWhoVSSqmwun1Y+IyPI2VHOFN5Bp9eWaKUUiF1+w7uk56TzHh1Rs3zWHsscY644C+n/3vQOmtZvCO+fvkGtnfanFF8pUop1XLdPixcdhf3TbqPsqqyRr+OlB0JXlZZRoWvolnHctgc/vCw1wZIk7+cwcEUa48N2kesPRbRS/66Ja/Pi8frocJb4f/uq6h9bH33eD1UeitrHjdWtsJbu8zjq31e/WW32XHYHDhtTpw2Z9Bjp93Z+Dpx4LSHWBf43B5mXcA+qtfZxa5//xHW7cMi3hnPvGHzWrRtla+K8qryesFypupMvWAJfF7uLQ9aVlpRSvGZ4uAyVeWYhm5WC0EQYh3BZ0X1znrqBFTdwGnsTMmul2aG5DO+0G+2via+WYcpG7TMV3/7Sm8lVW0wL1iMLYYYu//LZXfhsrtqHjttTuId8fRy9cJpd+L1ean0VVLlq6LSV0lZVRmnfKeo9FX6v6w6VXorg8pV+irb4CcemiC1oWIPETJ1AqfBdU0NtaYEXohQC1zX2f6nun1YtIbD5iAxJpHEmMQ237cxxh8qDQROQ8EUKrxOnjlZbx/NfYOJscWEPBuKdcQ2rSmukeY7p83Zok+FxpigN9qmfkJuqEzdN+2g7QPe2AOXVfla/0btsDn8b8622jfroO8OF8mu5JrngeXqlg3azuZqMAAClzttTmwS+e5LYwxVpqo2PLzBQRIULHUCp27oVPkaXtdYWFU/L60sbfQYgd8jxSa21oea9bx/Un+uP/v6iNUVNCw6LBGpeTONhEpvZf0zoLpfgQHlDR1YJ8pPUFRVFLTM4/U0qy52sdc7K4pzxGEXe82n6ZCfupvZDBiKTWz13lzrvqnGO+ODl4V4sw75Jt+EsjG2mE73CbOlRASn+N/o4ojM33VbM8Y0HGjhAi/E+qaGWtBjr/9xeVV5g6E6tNdQDQsVGU67kx72HvRw9WjzfXt93tqzosoQzXKNnAkFlvX6vCTHJDf8ybn6DbiBT+R1P2WHWu6w6b+AapiI+D/h2/XiFP1PUW3ObrOTYEsgwZlAJ/kAqZQKo9vfZ6GUUio8DQullFJhaVgopZQKS8NCKaVUWBoWSimlwtKwUEopFZaGhVJKqbA0LJRSSoUlxjR9sLqOTEQOA9+0Yhd9gCNtVJ1o6iqvA/S1dFRd5bV0ldcBrXstA40xqeEKdZmwaC0RWWOMGR/terRWV3kdoK+lo+oqr6WrvA5on9eizVBKKaXC0rBQSikVloZFrb9EuwJtpKu8DtDX0lF1ldfSVV4HtMNr0T4LpZRSYemZhVJKqbC6fViIyF9FpFhENke7Lq0hIv1F5BMR2SoiX4vIj6Ndp5YSkVgRWSUiG6zX8mC069QaImIXka9E5N1o16U1RGSPiGwSkfUisiba9WkNEekpIq+LyDbrf2ZKtOvUEiIy1Pp9VH+ViMgdETlWd2+GEpFzgVLgf40xw6Ndn5YSETfgNsasE5EkYC3wPWPMlihXrdnEPyF3gjGmVEScwBfAj40xK6NctRYRkZ8C44FkY8ycaNenpURkDzDeGNPp700QkeeBz40xz4hIDBBvjDkR7Xq1hojYgf3AJGNMa+45C6nbn1kYY5YCx6Jdj9YyxhQZY9ZZj08BW4F+0a1Vyxi/Uuup0/rqlJ9qRCQTuBh4Jtp1UX4ikgycCzwLYIyp6OxBYZkJ5EciKEDDoksSkSxgDPBldGvSclbTzXqgGFhsjOmsr+UJ4N8BX7Qr0gYM8KGIrBWRW6JdmVbIAQ4Dz1nNg8+ISEK0K9UG5gH/iNTONSy6GBFJBN4A7jDGlES7Pi1ljPEaY0YDmcBEEel0TYQiMgcoNsasjXZd2sg5xpixwEXAv1lNuJ2RAxgLPG2MGQOcBu6JbpVax2pKmwu8FqljaFh0IVb7/hvA340x/xft+rQFq3ngU2B2lKvSEucAc622/peB80TkxehWqeWMMQes78XAm8DE6NaoxQqBwoCz1dfxh0dndhGwzhhzKFIH0LDoIqxO4WeBrcaYx6Jdn9YQkVQR6Wk9jgPOB7ZFt1bNZ4y51xiTaYzJwt9E8LEx5tooV6tFRCTBunACq8nmAqBTXkFojDkI7BORodaimUCnuxCkjvlEsAkK/Kdj3ZqI/AOYDvQRkULgAWPMs9GtVYucA1wHbLLa+gH+wxizKIp1aik38Lx1dYcNeNUY06kvO+0C+gJv+j+T4ABeMsa8H90qtcr/A/5uNd8UADdGuT4tJiLxwCzghxE9Tne/dFYppVR42gyllFIqLA0LpZRSYWlYKKWUCkvDQimlVFgaFkoppcLSsFBKKRWWhoVS7cwa6rtPC7e9QUQy2mJfSjWHhoVSncsNQEa4Qkq1NQ0L1W2JSJY1+c0zIrJZRP4uIueLyDIR2SkiE62v5dbopMurh4gQkZ+KyF+txyOs7eMbOE6KiHxo7ePPgASsu9aa6Gm9iPzZumsdESkVkd+LyDoR+cgaAuVy/PNi/N0qH2ft5v9Z5TaJyLBI/sxU96Vhobq7QcCTwEhgGHA18C3gTuA/8I9Jda41OukvgIet7Z4ABonIpcBzwA+NMWcaOMYDwBfWPhYCAwBEJA+4Cv9orqMBL3CNtU0C/oHhxgKf4R+G5nVgDXCNMWa0MabMKnvEKve0VW+l2ly3HxtKdXu7jTGbAETka+AjY4wRkU1AFtAD/zhVg/HP5+AEMMb4ROQGYCPwZ2PMskaOcS7wfWu790TkuLV8JjAOWG2NuRSHf/4O8M9/8Yr1+EWgsVGEq9etrT6OUm1Nw0J1d56Ax76A5z78/x//CXxijLnUmlTq04Dyg/FPyduUPoRQg7AJ8Lwx5t4Wbl+tus5e9H9aRYg2QynVuB745zUGf+cyACLSA3/z1blAitWf0JClWM1LInIR0Mta/hFwuYikWet6i8hAa50NqN7n1fjnIQc4BSS14vUo1SIaFko17rfAb0RkGWAPWP448EdjzA7gJuCR6jf9EB4EzhWRdfjngdgLYIzZAtyPf6rSjcBi/MOzg3/2trNFZC1wHvAra/nfgD/V6eBWKuJ0iHKlOiARKTXGJEa7HkpV0zMLpZRSYemZhVJtRERuBH5cZ/EyY8y/RaM+SrUlDQullFJhaTOUUkqpsDQslFJKhaVhoZRSKiwNC6WUUmFpWCillArr/wM6T5X1XMg8HgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch2_1.best_score_, gsearch2_1.best_params_))\n",
    "test_means = gsearch2_1.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch2_1.cv_results_[ 'std_test_score' ]\n",
    "train_means = gsearch2_1.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = gsearch2_1.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch2_1.cv_results_).to_csv('my_preds_maxdepth_min_child_weights_1.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(max_depth), len(min_child_weight))\n",
    "train_scores = np.array(train_means).reshape(len(max_depth), len(min_child_weight))\n",
    "\n",
    "for i, value in enumerate(max_depth):\n",
    "    pyplot.plot(min_child_weight, -test_scores[i], label= 'test_max_depth:'   + str(value))\n",
    "#for i, value in enumerate(min_child_weight):\n",
    "#    pyplot.plot(max_depth, train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'min_child_weight' )                                                                                                      \n",
    "pyplot.ylabel( '- Log Loss' )\n",
    "pyplot.savefig('max_depth_vs_min_child_weght_1_test.png' )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
